• This is a workshop.
• But it is short.
• We will do some hands-on stuff but there is not a lot of time for practice.

You can find the slides (and additional materials) here: https://github.com/jstbcs/ReproducibleWorkflowWorkshop.

Julia Haaf

Assistant Professor at the Psychological Methods Department, University of Amsterdam

email: j.m.haaf@uva.nl

Job alert: Two temporary assistant professor positions at the Psychological Methods Unit

1. How can we minimize mistakes in psychological science?
2. What does it mean to have a fully reproducible pipeline?
3. What is git and how can I use it?
4. Using git for radically open data.

• Failures to replicate (e.g. Ebersole et al., 2016; Open Science Collaboration, 2015; Wagenmakers et al., 2016).
• Fraud (Bhattacharjee, 2013).
• Improbable findings have been published in top-tier journals (e.g. Bem, 2011).

• Change the incentive structure (e.g., Nosek et al., 2015; Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012).
• Be transparent and open (e.g. Rouder, 2016; Wicherts, Bakker, & Molenaar, 2011).
• Change the statistical approach (e.g. Benjamin et al., 2018; Erdfelder, 2010; Rouder et al., 2016)

• Change the incentive structure (e.g., Nosek et al., 2015; Wagenmakers et al., 2012).
• Be transparent and open (e.g. Rouder, 2016; Wicherts et al., 2011).
• Change the statistical approach (e.g. Benjamin et al., 2018; Erdfelder, 2010; Rouder et al., 2016)

Sources of mistakes:

• Errors when programming the experiment (e.g. randomization).
• Equipment failure (e.g. responses are collected unreliably).
• Lost data.
• Errors when coding the analysis (e.g. with data cleaning).
• Errors when reporting the analysis (e.g. typos).

Think about your own experience:

• Is there time pressure to collect data?
• Are there checks for coding experiments/surveys?
• Are there checks for running analyses?

• Prevalence: Roughly half the publications in 30 years of literature contained at least one malformed statement of a statistical test (Nuijten, Hartgerink, Assen, Epskamp, & Wicherts, 2016).
• Bias: Simple mistakes tend to go in researchers’ preferred direction (Gould, 1996).
• Persistence: Once in the literature mistakes are almost impossible to detect (Rouder, Haaf, & Snyder, 2019).

1. Sensitivity to operations: Focus on processes instead of outcomes.
2. Preoccupation with failure: Look for ways to proactively anticipate and avoid mistakes, and take small mistakes seriously.
3. Resilience in the face of failure and reluctance to simplify: In a resilient lab, when things go wrong — and they will — people talk about them, document them, and learn from them.
4. Deference to expertise: Each lab member has certain expertise.

1. Adopting a lab culture focused on learning from mistakes.
2. Implementing radical computer automation.
3. Standardizing organizational strategies across lab members.
4. Ensuring that statistical analyses are coded.
5. Developing expanded manuscripts in which documentation of analyses is woven into the manuscript files.

“An article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result.” Claerbout & Karrenbach, 1992

• Reproducible analysis.
• Reproducible graphs and tables.
• Reproducible numbers in text.

• Ideal: Code containerization.
• Minimal: Provide a list of packages and software needed (Open source!).
• Utopian: “I will be able to fully reproduce my analysis by 2035.”

git

• Versioning tool for collaboratively working on a product.
• Avoid retaining multiple versions of the same work product.
• ‘paper_final_final_B.docx’.
• Tutorial: Vuorre & Curley (n.d.).

If you need help installing some things from the instructions now is a good time.

link: https://github.com/jstbcs/ReproducibleWorkflowWorkshop#how-to-prepare

Version control:

• Version control is a system that records changes to a file or set of files over time so that you can recall specific versions later.
• History of all changes (who, what, when).
• Helps to avoid mistakes (working on the wrong version, deleting, …).
• Merging changes of multiple collaborators in one file.

• A lot! Which is why we only need a part of its functionality.
• Working on one product in (large) teams.
• Working on things that can break.
• git can only integrate and show changes in text files.
• binary files (images, etc.) can be tracked and uploaded but changes cannot be shown.
• Track changes for MS Word is improving.

Tools ➤ Global Options ➤ Git/SVN.

Make sure the first box is ticked and the “git.exe” (Windows) is included in the first box.

• Open the Terminal in R Studio.
• Set an email address and user name for git.

• Open the Terminal in R Studio.
• Set an email address and user name for git.

git config --global user.email "myemail@email.com"
git config --global user.name "My commit name"

File ➤ New Project ➤ Version Control ➤ Git

• You will have to type in you user name and password for github.
• Initializes a local git repository with an R project (opening a clean R Studio session when opeing).
• You can see the README file from github.
• Adds a .gitignore file.

• Make a new repository.
• Clone it using R Studio to make a local repository.

• Specifies intentionally untracked files to ignore.
• Each line in a gitignore file specifies a pattern.
• R Studio pre-specifies some useful patterns.
• For R Markdown: Cache files! .tiff, .eps, .rdb, .rdx

• Tell other people (and yourself in a year) why your project is useful, what they can do with your project, and how they can use it.
• On github default README files are Markdown files!

• Write a (short) README file for your test repository.
• Use Markdown formatting.

git add .gitignore myfirstrepo.Rproj

git can do autocomplete for file names!

git add .gitignore myfirstrepo.Rproj
git commit -am "My first commit"

Commits always have a commit message.

git add .gitignore myfirstrepo.Rproj
git commit -am "My first commit"
git push

Congrats! You have done it! Now local and remote repositories are up to date!

• Add your unstaged files.
• Commit the changes.
• Push to the remote repository.

Before you start working on the project the next time:

git pull

Pull, work some more, repeat.

git diff

• Remember: You cannot break things.
• Most likely you have a merge conflict.

• Resolve the conflict (Choose which changes to keep).
• Commit, and Push.

• Add, commit, push, pull.
• Use it!
• git documentation and error tracking are great!

• Standard: Data on request.
• Wicherts, Borsboom, Kats, & Molenaar (2006): Only 11 % of APA journal authors complied with initial request, an additional 16 % with repeated requests. A full 73 % of author teams never complied.
• More and more: Data on OSF.
• Needs to be hand-curated, only published when the paper is published.

• Rouder (2016)
• For people who are not meticulous.
• Commitment to radically open data: You can see all data, even the failed pilots.
• Nightly (or weekly) automatic upload to github.
• github.com/PerceptionCognitionLab

• Increases awareness to make good decisions.
• No data-management mistakes (no data versions).
• Automatic backup.
• Easy data sharing.
• Long-term availability.

• Shared local storage across experimental computers.
• Git or Github repository.
• Execution and scheduling

• Shared local storage across experimental computers.
• Git or Github repository.
• Execution and scheduling (task scheduler).

git add *.dat.*
git commit -m "automatic commit"
git push

• Not suitable for very large files (neuroimaging data, etc.).
• Not a properly curated archive.
• Github repositories can be connected to OSF.
• Privacy concerns.
• Being scooped.
• Being vulnerable.